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Abstract Body 
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Problem / Background / Context: 

Description of the problem addressed, prior research, and its intellectual context. 

Students’ basic reading and math skills can decline significantly during the summer 
months. Most attribute this to the prolonged period during which young students are 
outside of a structured learning environment (Cooper et al 1996). While summer learning 
loss occurs among all income groups, the effects are more pronounced for those from 
lower-income households than those from higher-income households where families have 
sufficient resources to provide enriching summer activities including camps, lessons, 
tutoring, art programs, and such (Entwisle, Alexander, and Olson 1997). These losses 
begin even in the earliest years of K-12 schooling and accumulate from year to year as 
students move through the crucial years of learning to read and have lasting impacts on 
student achievement (Alexander et al 2007). Moderate levels of evidence suggest that 
summer programs provided as school can ameliorate regression in achievement over the 
summer holidays (e.g. Borman and Dowling 2006). These findings have led to renewed 
interest in summer programs designed to mitigate, or even reverse, the trend of summer 
learning loss. 


Purpose / Objective / Research Question / Focus of Research: 

Description of the focus of the research. 

In this paper, we evaluate the effectiveness of a novel summer program run by the State of New 
Mexico known as K-3 Plus. We seek to evaluate the extent to which participation in the K-3 Plus 
summer session reduces or even reverses the trend of summer learning loss. Using funding from 
a 2010 Investing in Innovation validation grant, we established the Starts mart K-3 Plus program, 
a program designed to mimic the standard state-funded K-3 Plus program except it is funded by 
our grant and requires randomization into intervention and control groups. The multi-site 
randomized controlled trial design allows us to rigorously evaluate the question of whether 
program participation improves academic achievement. 

Improvement Initiative / Intervention / Program / Practice: 

Description of the improvement initiative or related intervention, program, or practice. 

The New Mexico State Legislature originally created the K-3 Plus program as a pilot program. 
The StartSmart K-3 Plus program is implemented with high fidelity to the state-funded model, 
which requires 25 days of additional instruction each summer prior to grades K-3 (though 
participation is voluntary rather than compulsory) in smaller class sizes and includes instruction 
from certified teachers who have completed professional development, standard school meal 
service (breakfast and lunch), transportation, and a parent involvement component. Because the 
problem of summer learning loss is more pronounced among low-SES populations, state-funded 
K-3 Plus is provided in schools where 85% or more of students qualify for free or reduced-price 
lunches (FRL) under federal standards. In the StartSmart K-3 Plus program, we extend the 
program to include schools with 70% more of their students qualifying under federal FRL 
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standards. In contrast to many summer programs, the instruction is not focused strictly on 
remediation but instead begins the regular curriculum earlier. 

Setting: 

Description of the research location and partners involved, if applicable. 

The StartSmart K-3 Plus study is based in 7 districts around New Mexico, specifically 
Albuquerque Public Schools, Gallup-McKinley Public Schools, Santa Fe Public Schools, 
Roswell Independent School District, Hobbs Municipal Schools, Deming Public Schools, and 
Gadsden Independent School District. Within these districts, only schools that recruited 
sufficient numbers of student participants and met the necessary criteria (70% FRL or higher) 
participate in the program. Summer services are provided in these partnering districts. 

Perhaps related to the high-poverty nature of the population served by StartSmart 
schools, our student population is highly mobile. Students who began the study in a participating 
school but moved to another district (or a non-participating school within a participating district) 
are tracked and followed. Where possible, intervention group students are invited to attend a 
summer session at the nearest available school. 

Population / Participants / Subjects: 

Description of the participants in the research: who, how many, key features, or characteristics. 

Subjects were recruited in two cohorts in the spring prior to enrollment in Kindergarten, with the 
first cohort enrolling in the Spring of 201 1 (n = 547) to begin Kindergarten in the Fall of 201 1, 
and the larger second cohort beginning in the Spring of 2012 (n = 1001). Subjects were recruited 
primarily from Kindergarten registration events, but the project also advertised in K-12 schools, 
day cares, Head Start programs, and other community venues. Subjects were advised that if they 
enroll in the program they would receive $50 for each assessment they complete as well as 
having about a 55% chance of having their child receive summer services at no charge. 
Intervention students are not compelled to attend summer services. In their first year in the 
program, about 80% of intervention students conform to their treatment assignment, though this 
figure drops to 60-70% in subsequent years. 

Research Design: 

Description of the research design. 

Our project employs a multi-site randomized controlled trial design. Students were recruited and 
then randomly assigned to intervention or control within each school. About 55% of students 
were assigned to the intervention group, which receives summer services in addition to regular 
school year services, and about 45% of students were assigned to the control group, which 
receives regular school year services only. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data or use of existing databases. 

Students in both treatment conditions are assessed at baseline in the Spring before they begin 
Kindergarten. The students are then assessed each Fall and Spring thereafter (following the 
summer session but within the first 6 weeks of school for the Fall and within the last 6 weeks of 
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school in the Spring). Students are assessed using the Woodcock Johnson battery (or 
Woodcock-Munoz for students whose primary language is Spanish), the Peabody Picture 
Vocabulary test (or Test de Vocabulario en Imagenes Peabody for Spanish-speakers), and the 
SSIS social skills measurement instrument. In terms of measures of achievement, we are 
interested in 6 outcome domains: Expressive Vocabulary, Reading, Math, Writing, Receptive 
Language, and Social Skills. 

Participating students’ parents or guardians are also asked to complete a survey to collect 
relevant infonnation about socio-demographic characteristics of their household and their 
attitudes about their child’s education. 

We evaluate the effect of the K-3 Plus program using a multi-level model appropriate for a 
multi-site randomized controlled trial like ours (see Raudenbush and Liu 2000). The outcome 
measure for each outcome domain named above serves as the dependent variable with treatment 
assignment, maternal education, gender, and baseline achievement at independent variables. 
Random effects capture effect heterogeneity across sites. 

Findings / Outcomes: 

Description of the main findings or outcomes, with specific details. 

In this abstract, we include intent-to-treat effects for the effects of K-3 Plus prior to Kindergarten 
on student achievement at the beginning of Kindergarten, interpreting these as effects of program 
participation on Kindergarten readiness in the most conservative way (versus treatment-on-the- 
treated). In the paper to be presented at the conference, we will share additional findings on the 
effect of the program through the first grade year. The average treatment effects of program 
participation appear in Table 1 below (values of control variables are not shown). 

(Table 1 about here) 

The results show that across four outcome domains of interest — Expressive Vocabulary, Letter- 
Word ID, Applied Problems, and Basic Writing — students randomly assigned to spend the 
summer in K-3 Plus outperfonned those who were randomly assigned to receive regular school 
year services only. We interpret this to mean that K-3 Plus students were better prepared for 
Kindergarten at the beginning of the school year than students who did not have K-3 Plus. These 
preliminary ITT estimates find no statistically significant effect of K-3 Plus on Social Skills or 
Receptive Language. 

Conclusions: 

Description of conclusions, recommendations, and limitations, based on findings. 

Preliminary results shared here show that the K-3 Plus program is an effective way to improve 
students’ preparedness for Kindergarten. Preliminary analyses based only on our smaller 1 st 
cohort’s performance in 1 st grade (not shown here but to be presented at the conference 
supplemented with results from the 2 nd cohort) suggest that these gains in achievement continue 
with subsequent participation in the program. Results, while preliminary, suggest that the K-3 
Plus model is an effective way to improve student achievement in high-poverty schools by 
ameliorating summer learning loss. 
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Appendices 

Not included in page count. 
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Appendix B. Tables and Figures 

Not included in page count. 


Table 1: ITT Effects of StartSmart K-3 Plus on Student Achievement at the Beginning of 

Kindergarten 


Outcome 

Domain 

Baseline 

Equivalence 

Supported? 

Avg. Treatment Effect 
(ITT) 

n 

Expressive 

Yes 

1.638* 

1572 

Vocabulary 


(.582) 


Letter- Word ID 

Yes 

4.115* 

1586 



(.529) 


Applied 

Yes 

1.803* 

1542 

Problems 


(.414) 


Basic Writing 

Yes 

4.723* 

1471 



(.610) 


Social Skills 

Yes 

0.064 

1567 



(.663) 


Receptive 

Yes 

.732 

1095 

Language 


(.509) 



Note: Standard Errors appear below estimates in parentheses. Baseline Equivalence is supported 
if we fail to reject the null hypothesis of no difference between groups at the p < .05 level of 
confidence on a two-tailed test. * denotes p < .05, two-tailed. 
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